

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>归置组排障 &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/css/custom.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/jquery.js"></script>
        <script src="../../../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script data-url_root="../../../" id="documentation_options" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/doctools.js"></script>
        <script src="../../../_static/sphinx_highlight.js"></script>
    
    <script type="text/javascript" src="../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../genindex/" />
    <link rel="search" title="Search" href="../../../search/" />
    <link rel="next" title="日志记录和调试" href="../log-and-debug/" />
    <link rel="prev" title="OSD 故障排除" href="../troubleshooting-osd/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../../" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="../../">Ceph 存储集群</a></li>
          <li class="breadcrumb-item"><a href="../../operations/">集群运维</a></li>
      <li class="breadcrumb-item active">归置组排障</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../../../_sources/rados/troubleshooting/troubleshooting-pg.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../" class="icon icon-home"> Ceph
          

          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../start/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephadm/">Cephadm</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../">Ceph 存储集群</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../configuration/">配置</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../operations/">运维</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../man/">    手册页</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">故障排除</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../community/">Ceph 社区</a></li>
<li class="toctree-l3"><a class="reference internal" href="../log-and-debug/">日志记录和调试</a></li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting-mon/">监视器故障排除</a></li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting-osd/">OSD 故障排除</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">归置组排障</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#id2">归置组总不整洁</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id5">卡住的归置组</a></li>
<li class="toctree-l4"><a class="reference internal" href="#failures-osd-peering">归置组挂了——互联失败</a></li>
<li class="toctree-l4"><a class="reference internal" href="#failures-osd-unfound">未找到的对象</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id8">无根归置组</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id9">只有几个 OSD 接收数据</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id10">不能写入数据</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id11">归置组不一致</a></li>
<li class="toctree-l4"><a class="reference internal" href="#active-clean">纠删编码的归置组不是 active+clean</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../memory-profiling/">内存剖析</a></li>
<li class="toctree-l3"><a class="reference internal" href="../cpu-profiling/">CPU 剖析</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../api/">APIs</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../monitoring/">监控概览</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../hardware-monitoring/">硬件监控</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <section id="id1">
<h1>归置组排障<a class="headerlink" href="#id1" title="Permalink to this heading"></a></h1>
<section id="id2">
<h2>归置组总不整洁<a class="headerlink" href="#id2" title="Permalink to this heading"></a></h2>
<p>归置组（ PG ）们应该保持在 <code class="docutils literal notranslate"><span class="pre">active</span></code> 状态、 <code class="docutils literal notranslate"><span class="pre">active+remapped</span></code> 状态或者
<code class="docutils literal notranslate"><span class="pre">active+degraded</span></code> 状态，一直不能进入 <code class="docutils literal notranslate"><span class="pre">active+clean</span></code> 状态可能预示着
Ceph 集群的配置有问题。</p>
<p>在这种情况下，应该仔细检查下<a class="reference external" href="../../configuration/pool-pg-config-ref">存储池、归置组和 CRUSH 配置参考</a>
里的配置选项，做一些调整。</p>
<p>一般来说，你的集群应该有一个以上 OSD 、并且存储池的对象副本数大于 2 。</p>
<section id="one-node-cluster">
<span id="id3"></span><h3>单节点集群<a class="headerlink" href="#one-node-cluster" title="Permalink to this heading"></a></h3>
<p>Ceph 不再提供单节点的运营文档，
因为你不可能在单个节点上设计、部署一套分布式计算系统。
另外，在运行 Ceph 守护进程的节点上同时用内核客户端挂载会导致死机，
这是 Linux 内核自身的问题（除非你在 VM 里跑客户端）。
抛开这里提到的诸多限制，用单节点配置体验 Ceph 是可以的。</p>
<p>如果你正尝试在单个节点上创建一个集群，在创建监视器和 OSD 前，
必须把配置文件里的 <code class="docutils literal notranslate"><span class="pre">osd</span> <span class="pre">crush</span> <span class="pre">chooseleaf</span> <span class="pre">type</span></code> 的默认值从
<code class="docutils literal notranslate"><span class="pre">1</span></code> （表示 <code class="docutils literal notranslate"><span class="pre">host</span></code> 或 <code class="docutils literal notranslate"><span class="pre">node</span></code> ）改成 <code class="docutils literal notranslate"><span class="pre">0</span></code> （表示 <code class="docutils literal notranslate"><span class="pre">osd</span></code> ）。
就是告诉 Ceph ， OSD 可以和同一主机上的其它 OSD 互联。
如果你想要建一套单节点集群，
而 <code class="docutils literal notranslate"><span class="pre">osd</span> <span class="pre">crush</span> <span class="pre">chooseleaf</span> <span class="pre">type</span></code> 却大于 <code class="docutils literal notranslate"><span class="pre">0</span></code> ，
根据你的配置， Ceph 就会让一个 OSD 上的 PG 与另一节点、机箱、机架、行、甚至是数据中心里面的 PG 互联。</p>
<div class="admonition tip">
<p class="admonition-title">Tip</p>
<p><strong>不要</strong> 在运行 Ceph 存储集群的同一主机上用内核客户端直接挂载，因为会有内核冲突。
但是，你可以在同一节点上的虚拟机里面用内核客户端挂载。</p>
</div>
<p>如果你在用单个磁盘创建 OSD ，
你必须先创建好数据目录。</p>
</section>
<section id="osd">
<h3>OSD 数量小于副本数<a class="headerlink" href="#osd" title="Permalink to this heading"></a></h3>
<p>如果你已经把两个 OSD 带到了 <code class="docutils literal notranslate"><span class="pre">up</span></code> 且 <code class="docutils literal notranslate"><span class="pre">in</span></code> 状态，
却还没见到处于 <code class="docutils literal notranslate"><span class="pre">active</span> <span class="pre">+</span> <span class="pre">clean</span></code> 状态的归置组，
可能是因为你的 <code class="docutils literal notranslate"><span class="pre">osd</span> <span class="pre">pool</span> <span class="pre">default</span> <span class="pre">size</span></code> 大于 <code class="docutils literal notranslate"><span class="pre">2</span></code> 。</p>
<p>有几种办法解决这个问题。如果你想让一个双副本集群进入 <code class="docutils literal notranslate"><span class="pre">active</span> <span class="pre">+</span> <span class="pre">degraded</span></code> 状态，
你可以把 <code class="docutils literal notranslate"><span class="pre">osd</span> <span class="pre">pool</span> <span class="pre">default</span> <span class="pre">min</span> <span class="pre">size</span></code> 设置为 <code class="docutils literal notranslate"><span class="pre">2</span></code> ，
这样就能在 <code class="docutils literal notranslate"><span class="pre">active</span> <span class="pre">+</span> <span class="pre">degraded</span></code> 状态下写入对象了。
你还可以把 <code class="docutils literal notranslate"><span class="pre">osd</span> <span class="pre">pool</span> <span class="pre">default</span> <span class="pre">size</span></code> 设置成 <code class="docutils literal notranslate"><span class="pre">2</span></code> ，
这样你就有两份副本了（原始的和一个副本），
这种情况下，集群还能达到 <code class="docutils literal notranslate"><span class="pre">active</span> <span class="pre">+</span> <span class="pre">clean</span></code> 状态。</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>你可以在运行时做出更改。如果在配置文件里更改，
就需要重启集群。</p>
</div>
</section>
<section id="id4">
<h3>存储池副本数为 1<a class="headerlink" href="#id4" title="Permalink to this heading"></a></h3>
<p>如果把 <code class="docutils literal notranslate"><span class="pre">osd</span> <span class="pre">pool</span> <span class="pre">default</span> <span class="pre">size</span></code> 设置成 <code class="docutils literal notranslate"><span class="pre">1</span></code> ，你就只有一份对象副本。
OSD 靠其它 OSD 们告诉它，它应该持有哪些对象。
如果第一个 OSD 有某个对象的一个副本、并且没有第二个副本，
那么就没有第二个 OSD 能告诉第一个 OSD 它应该有那个副本。
对于映射到第一个 OSD 的每个归置组（见 <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">pg</span> <span class="pre">dump</span></code> ），
你可以强制第一个 OSD 通知它需要的归置组，用命令：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt1:before {
  content: "$ ";
}
</style><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>force-create-pg<span class="w"> </span>&lt;pgid&gt;</span>
</pre></div></div></section>
<section id="crush">
<h3>CRUSH 图错误<a class="headerlink" href="#crush" title="Permalink to this heading"></a></h3>
<p>归置组仍然不整洁的另一个原因可能是 CRUSH 图中的错误。</p>
</section>
</section>
<section id="id5">
<h2>卡住的归置组<a class="headerlink" href="#id5" title="Permalink to this heading"></a></h2>
<p>有失败时归置组会进入“degraded”（降级）或“peering”（连接建立中）状态，
这事时有发生，通常这些状态意味着正常的失败恢复正在进行。
然而，如果一个归置组长时间处于某个这些状态就意味着有更大的问题，
因此监视器在归置组卡 （stuck） 在非最优状态时会警告。
我们具体检查：</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">inactive</span></code> （不活跃）——归置组长时间无活跃
（即它不能提供读写服务了）；</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">unclean</span></code> （不干净）——归置组长时间不干净
（例如它未能从前面的失败完全恢复）；</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">stale</span></code> （不新鲜）——归置组状态没有被 <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code> 更新，
表明存储这个归置组的所有节点可能都挂了。</p></li>
</ul>
<p>你可以摆出卡住的归置组，用下列命令之一：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>pg<span class="w"> </span>dump_stuck<span class="w"> </span>stale</span>
<span class="prompt1">ceph<span class="w"> </span>pg<span class="w"> </span>dump_stuck<span class="w"> </span>inactive</span>
<span class="prompt1">ceph<span class="w"> </span>pg<span class="w"> </span>dump_stuck<span class="w"> </span>unclean</span>
</pre></div></div><p>卡在 <code class="docutils literal notranslate"><span class="pre">stale</span></code> 状态的归置组通过修复 <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code> 进程通常可以修复；
卡在 <code class="docutils literal notranslate"><span class="pre">inactive</span></code> 状态的归置组通常是互联问题
（参见 <a class="reference internal" href="#failures-osd-peering"><span class="std std-ref">归置组挂了——互联失败</span></a> ）；
卡在 <code class="docutils literal notranslate"><span class="pre">unclean</span></code> 状态的归置组通常是由于某些原因阻止了恢复的完成，像未找到的对象
（参见 <a class="reference internal" href="#failures-osd-unfound"><span class="std std-ref">未找到的对象</span></a> ）。</p>
</section>
<section id="failures-osd-peering">
<span id="id6"></span><h2>归置组挂了——互联失败<a class="headerlink" href="#failures-osd-peering" title="Permalink to this heading"></a></h2>
<p>在某些情况下， <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code> 的 <cite>互联</cite> 进程会遇到问题，
使 PG 不能活跃、可用。
例如 <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">health</span></code> 也许显示：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>health<span class="w"> </span>detail</span>
</pre></div></div><div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">HEALTH_ERR</span> <span class="mi">7</span> <span class="n">pgs</span> <span class="n">degraded</span><span class="p">;</span> <span class="mi">12</span> <span class="n">pgs</span> <span class="n">down</span><span class="p">;</span> <span class="mi">12</span> <span class="n">pgs</span> <span class="n">peering</span><span class="p">;</span> <span class="mi">1</span> <span class="n">pgs</span> <span class="n">recovering</span><span class="p">;</span> <span class="mi">6</span> <span class="n">pgs</span> <span class="n">stuck</span> <span class="n">unclean</span><span class="p">;</span> <span class="mi">114</span><span class="o">/</span><span class="mi">3300</span> <span class="n">degraded</span> <span class="p">(</span><span class="mf">3.455</span><span class="o">%</span><span class="p">);</span> <span class="mi">1</span><span class="o">/</span><span class="mi">3</span> <span class="ow">in</span> <span class="n">osds</span> <span class="n">are</span> <span class="n">down</span>
<span class="o">...</span>
<span class="n">pg</span> <span class="mf">0.5</span> <span class="ow">is</span> <span class="n">down</span><span class="o">+</span><span class="n">peering</span>
<span class="n">pg</span> <span class="mf">1.4</span> <span class="ow">is</span> <span class="n">down</span><span class="o">+</span><span class="n">peering</span>
<span class="o">...</span>
<span class="n">osd</span><span class="mf">.1</span> <span class="ow">is</span> <span class="n">down</span> <span class="n">since</span> <span class="n">epoch</span> <span class="mi">69</span><span class="p">,</span> <span class="n">last</span> <span class="n">address</span> <span class="mf">192.168.106.220</span><span class="p">:</span><span class="mi">6801</span><span class="o">/</span><span class="mi">8651</span>
</pre></div>
</div>
<p>查询下集群，看看到底 PG 为什么被标记为 <code class="docutils literal notranslate"><span class="pre">down</span></code> ，按下面的格式运行命令：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>pg<span class="w"> </span><span class="m">0</span>.5<span class="w"> </span>query</span>
</pre></div></div><div class="highlight-javascript notranslate"><div class="highlight"><pre><span></span><span class="p">{</span><span class="w"> </span><span class="s2">&quot;state&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;down+peering&quot;</span><span class="p">,</span>
<span class="w">  </span><span class="p">...</span>
<span class="w">  </span><span class="s2">&quot;recovery_state&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">       </span><span class="p">{</span><span class="w"> </span><span class="s2">&quot;name&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;Started\/Primary\/Peering\/GetInfo&quot;</span><span class="p">,</span>
<span class="w">         </span><span class="s2">&quot;enter_time&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;2012-03-06 14:40:16.169679&quot;</span><span class="p">,</span>
<span class="w">         </span><span class="s2">&quot;requested_info_from&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[]},</span>
<span class="w">       </span><span class="p">{</span><span class="w"> </span><span class="s2">&quot;name&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;Started\/Primary\/Peering&quot;</span><span class="p">,</span>
<span class="w">         </span><span class="s2">&quot;enter_time&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;2012-03-06 14:40:16.169659&quot;</span><span class="p">,</span>
<span class="w">         </span><span class="s2">&quot;probing_osds&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">               </span><span class="mf">0</span><span class="p">,</span>
<span class="w">               </span><span class="mf">1</span><span class="p">],</span>
<span class="w">         </span><span class="s2">&quot;blocked&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;peering is blocked due to down osds&quot;</span><span class="p">,</span>
<span class="w">         </span><span class="s2">&quot;down_osds_we_would_probe&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">               </span><span class="mf">1</span><span class="p">],</span>
<span class="w">         </span><span class="s2">&quot;peering_blocked_by&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">               </span><span class="p">{</span><span class="w"> </span><span class="s2">&quot;osd&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">1</span><span class="p">,</span>
<span class="w">                 </span><span class="s2">&quot;current_lost_at&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">0</span><span class="p">,</span>
<span class="w">                 </span><span class="s2">&quot;comment&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;starting or marking this osd lost may let us proceed&quot;</span><span class="p">}]},</span>
<span class="w">       </span><span class="p">{</span><span class="w"> </span><span class="s2">&quot;name&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;Started&quot;</span><span class="p">,</span>
<span class="w">         </span><span class="s2">&quot;enter_time&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;2012-03-06 14:40:16.169513&quot;</span><span class="p">}</span>
<span class="w">   </span><span class="p">]</span>
<span class="p">}</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">recovery_state</span></code> 段告诉我们因 <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code> 守护进程挂了而导致互联被阻塞，本例是 <code class="docutils literal notranslate"><span class="pre">osd.1</span></code> 挂了，
启动这个 <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code> 应该就可以恢复。</p>
<p>另外，如果 <code class="docutils literal notranslate"><span class="pre">osd.1</span></code> 是灾难性的失败（如硬盘损坏），
我们可以告诉集群它丢失（ <code class="docutils literal notranslate"><span class="pre">lost</span></code> ）了，
让集群尽力完成副本拷贝。</p>
<div class="admonition important">
<p class="admonition-title">Important</p>
<p>此信息告知集群说一个 OSD 丢失了，非常危险，
因为集群不能保证其它数据副本是一致且最新！</p>
</div>
<p>要报告一个 OSD 丢失（ <code class="docutils literal notranslate"><span class="pre">lost</span></code> ）了，并且让 Ceph 无论如何继续恢复，
按下面的格式执行：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>lost<span class="w"> </span><span class="m">1</span></span>
</pre></div></div><p>恢复将继续。</p>
</section>
<section id="failures-osd-unfound">
<span id="id7"></span><h2>未找到的对象<a class="headerlink" href="#failures-osd-unfound" title="Permalink to this heading"></a></h2>
<p>某几种失败相组合可能导致 Ceph 抱怨有找不到（ <code class="docutils literal notranslate"><span class="pre">unfound</span></code> ）的对象：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>health<span class="w"> </span>detail</span>
</pre></div></div><div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">HEALTH_WARN</span> <span class="mi">1</span> <span class="n">pgs</span> <span class="n">degraded</span><span class="p">;</span> <span class="mi">78</span><span class="o">/</span><span class="mi">3778</span> <span class="n">unfound</span> <span class="p">(</span><span class="mf">2.065</span><span class="o">%</span><span class="p">)</span>
<span class="n">pg</span> <span class="mf">2.4</span> <span class="ow">is</span> <span class="n">active</span><span class="o">+</span><span class="n">degraded</span><span class="p">,</span> <span class="mi">78</span> <span class="n">unfound</span>
</pre></div>
</div>
<p>这意味着存储集群知道一些对象
（或者现有对象的较新副本）存在，却没有找到它们的副本。下例展示了这种情况是如何发生的，
一个 PG 的数据存储在 ceph-osd 1 和 2 上：</p>
<ul class="simple">
<li><p>1 挂了；</p></li>
<li><p>2 独自处理一些写动作；</p></li>
<li><p>1 起来了；</p></li>
<li><p>1 和 2 重新互联， 1 上面丢失的对象加入队列准备恢复；</p></li>
<li><p>新对象还未拷贝完， 2 挂了。</p></li>
</ul>
<p>这时， 1 知道这些对象存在，但是活着的 <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code> 都没有副本，这种情况下，读写这些对象的 IO 就会被阻塞，
集群只能指望节点早点恢复。
这时我们假设用户希望先得到一个 IO 错误。</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>上面描述的情境是可能导致数据丢失的其中一个原因，就是
在给多副本存储池配置 <code class="docutils literal notranslate"><span class="pre">size=2</span></code> ，
或者给纠删码存储池配置 <code class="docutils literal notranslate"><span class="pre">m=1</span></code> 。</p>
</div>
<p>确认哪些对象找不到了，命令格式如下：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>pg<span class="w"> </span><span class="m">2</span>.4<span class="w"> </span>list_unfound<span class="w"> </span><span class="o">[</span>starting<span class="w"> </span>offset,<span class="w"> </span><span class="k">in</span><span class="w"> </span>json<span class="o">]</span></span>
</pre></div></div><div class="highlight-javascript notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
<span class="w">  </span><span class="s2">&quot;num_missing&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">1</span><span class="p">,</span>
<span class="w">  </span><span class="s2">&quot;num_unfound&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">1</span><span class="p">,</span>
<span class="w">  </span><span class="s2">&quot;objects&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">      </span><span class="p">{</span>
<span class="w">          </span><span class="s2">&quot;oid&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">{</span>
<span class="w">              </span><span class="s2">&quot;oid&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;object&quot;</span><span class="p">,</span>
<span class="w">              </span><span class="s2">&quot;key&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;&quot;</span><span class="p">,</span>
<span class="w">              </span><span class="s2">&quot;snapid&quot;</span><span class="o">:</span><span class="w"> </span><span class="o">-</span><span class="mf">2</span><span class="p">,</span>
<span class="w">              </span><span class="s2">&quot;hash&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">2249616407</span><span class="p">,</span>
<span class="w">              </span><span class="s2">&quot;max&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">0</span><span class="p">,</span>
<span class="w">              </span><span class="s2">&quot;pool&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">2</span><span class="p">,</span>
<span class="w">              </span><span class="s2">&quot;namespace&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;&quot;</span>
<span class="w">          </span><span class="p">},</span>
<span class="w">          </span><span class="s2">&quot;need&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;43&#39;251&quot;</span><span class="p">,</span>
<span class="w">          </span><span class="s2">&quot;have&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;0&#39;0&quot;</span><span class="p">,</span>
<span class="w">          </span><span class="s2">&quot;flags&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;none&quot;</span><span class="p">,</span>
<span class="w">          </span><span class="s2">&quot;clean_regions&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;clean_offsets: [], clean_omap: 0, new_object: 1&quot;</span><span class="p">,</span>
<span class="w">          </span><span class="s2">&quot;locations&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">              </span><span class="s2">&quot;0(3)&quot;</span><span class="p">,</span>
<span class="w">              </span><span class="s2">&quot;4(2)&quot;</span>
<span class="w">          </span><span class="p">]</span>
<span class="w">      </span><span class="p">}</span>
<span class="w">  </span><span class="p">],</span>
<span class="w">  </span><span class="s2">&quot;state&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;NotRecovering&quot;</span><span class="p">,</span>
<span class="w">  </span><span class="s2">&quot;available_might_have_unfound&quot;</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w">  </span><span class="s2">&quot;might_have_unfound&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">      </span><span class="p">{</span>
<span class="w">          </span><span class="s2">&quot;osd&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;2(4)&quot;</span><span class="p">,</span>
<span class="w">          </span><span class="s2">&quot;status&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;osd is down&quot;</span>
<span class="w">      </span><span class="p">}</span>
<span class="w">  </span><span class="p">],</span>
<span class="w">  </span><span class="s2">&quot;more&quot;</span><span class="o">:</span><span class="w"> </span><span class="kc">false</span>
<span class="p">}</span>
</pre></div>
</div>
<p>如果在一次查询里列出的对象太多，
<code class="docutils literal notranslate"><span class="pre">more</span></code> 这个字段将为 <code class="docutils literal notranslate"><span class="pre">true</span></code> ，因此你可以查询更多。
（命令行工具可能隐藏了，但这里没有）</p>
<p>然后找出哪些 OSD 上探测到了、
或可能包含数据。</p>
<p>在这份罗列结果的末尾（ <code class="docutils literal notranslate"><span class="pre">more:</span> <span class="pre">false</span></code> 之前），
<code class="docutils literal notranslate"><span class="pre">available_might_have_unfound</span></code> 是 true 的时候会有 <code class="docutils literal notranslate"><span class="pre">might_have_unfound</span></code> 。
这个和 <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">pg</span> <span class="pre">#.#</span> <span class="pre">query</span></code> 的输出是等价的，
只是这个结果让我们省去了直接查询（ <code class="docutils literal notranslate"><span class="pre">query</span></code> ）的必要。
<code class="docutils literal notranslate"><span class="pre">might_have_unfound</span></code> 信息的提供方式和下文 <code class="docutils literal notranslate"><span class="pre">query</span></code> 的相同，
仅有的差异是有 <code class="docutils literal notranslate"><span class="pre">already</span> <span class="pre">probed</span></code> 状态的 OSD 会被忽略。</p>
<p><code class="docutils literal notranslate"><span class="pre">query</span></code> 的用法：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>pg<span class="w"> </span><span class="m">2</span>.4<span class="w"> </span>query</span>
</pre></div></div><div class="highlight-javascript notranslate"><div class="highlight"><pre><span></span><span class="s2">&quot;recovery_state&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">     </span><span class="p">{</span><span class="w"> </span><span class="s2">&quot;name&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;Started\/Primary\/Active&quot;</span><span class="p">,</span>
<span class="w">       </span><span class="s2">&quot;enter_time&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;2012-03-06 15:15:46.713212&quot;</span><span class="p">,</span>
<span class="w">       </span><span class="s2">&quot;might_have_unfound&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">             </span><span class="p">{</span><span class="w"> </span><span class="s2">&quot;osd&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">1</span><span class="p">,</span>
<span class="w">               </span><span class="s2">&quot;status&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;osd is down&quot;</span><span class="p">}]},</span>
</pre></div>
</div>
<p>本案例中，集群知道 <code class="docutils literal notranslate"><span class="pre">osd.1</span></code> 可能有数据，但它挂了（ <code class="docutils literal notranslate"><span class="pre">down</span></code> ）。所有可能的状态有：</p>
<ul class="simple">
<li><p>已经探测到了</p></li>
<li><p>在查询</p></li>
<li><p>OSD 挂了</p></li>
<li><p>尚未查询</p></li>
</ul>
<p>有时候集群要花一些时间来查询可能的位置。</p>
<p>还有一种可能性，对象存在于其它位置却未被列出，
例如，集群里的一个 <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code> 停止且被剔出，
然后完全恢复了；后来的失败、恢复后仍有未找到的对象，
它也不会觉得早已死亡的 <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code> 上仍可能包含这些对象。
（这种情况几乎不太可能发生）。</p>
<p>如果所有位置都查询过了仍有对象丢失，
那就得放弃丢失的对象了。
这仍可能是罕见的失败组合导致的，
集群在写入完成前，未能得知写入是否已执行。
以下命令把未找到的（ unfound ）对象标记为丢失（ lost ）。</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>pg<span class="w"> </span><span class="m">2</span>.5<span class="w"> </span>mark_unfound_lost<span class="w"> </span>revert<span class="p">|</span>delete</span>
</pre></div></div><p>上述最后一个参数 （ <code class="docutils literal notranslate"><span class="pre">revert|delete</span></code> ）告诉集群应该
如何处理丢失的对象。</p>
<p><code class="docutils literal notranslate"><span class="pre">delete</span></code> 选项将导致完全删除它们。</p>
<p><code class="docutils literal notranslate"><span class="pre">revert</span></code> 选项（纠删码存储池不可用）会回滚到前一个版本或者
（如果它是新对象的话）删除它。要慎用，
它可能迷惑那些期望对象存在的应用程序。</p>
</section>
<section id="id8">
<h2>无根归置组<a class="headerlink" href="#id8" title="Permalink to this heading"></a></h2>
<p>拥有归置组拷贝的 OSD 都可以失败，
在这种情况下，那一部分的对象存储不可用，
监视器就不会收到那些归置组的状态更新了。
为检测这种情况，监视器把任何主 OSD 失败的归置组标记为 <code class="docutils literal notranslate"><span class="pre">stale</span></code> （不新鲜），例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>health</span>
</pre></div></div><div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">HEALTH_WARN</span> <span class="mi">24</span> <span class="n">pgs</span> <span class="n">stale</span><span class="p">;</span> <span class="mi">3</span><span class="o">/</span><span class="mi">300</span> <span class="ow">in</span> <span class="n">osds</span> <span class="n">are</span> <span class="n">down</span>
</pre></div>
</div>
<p>你能找出哪些归置组 <code class="docutils literal notranslate"><span class="pre">stale</span></code> 、和最后存储这些归置组的 OSD ，
命令如下：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>health<span class="w"> </span>detail</span>
</pre></div></div><div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">HEALTH_WARN</span> <span class="mi">24</span> <span class="n">pgs</span> <span class="n">stale</span><span class="p">;</span> <span class="mi">3</span><span class="o">/</span><span class="mi">300</span> <span class="ow">in</span> <span class="n">osds</span> <span class="n">are</span> <span class="n">down</span>
<span class="o">...</span>
<span class="n">pg</span> <span class="mf">2.5</span> <span class="ow">is</span> <span class="n">stuck</span> <span class="n">stale</span><span class="o">+</span><span class="n">active</span><span class="o">+</span><span class="n">remapped</span><span class="p">,</span> <span class="n">last</span> <span class="n">acting</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">0</span><span class="p">]</span>
<span class="o">...</span>
<span class="n">osd</span><span class="mf">.10</span> <span class="ow">is</span> <span class="n">down</span> <span class="n">since</span> <span class="n">epoch</span> <span class="mi">23</span><span class="p">,</span> <span class="n">last</span> <span class="n">address</span> <span class="mf">192.168.106.220</span><span class="p">:</span><span class="mi">6800</span><span class="o">/</span><span class="mi">11080</span>
<span class="n">osd</span><span class="mf">.11</span> <span class="ow">is</span> <span class="n">down</span> <span class="n">since</span> <span class="n">epoch</span> <span class="mi">13</span><span class="p">,</span> <span class="n">last</span> <span class="n">address</span> <span class="mf">192.168.106.220</span><span class="p">:</span><span class="mi">6803</span><span class="o">/</span><span class="mi">11539</span>
<span class="n">osd</span><span class="mf">.12</span> <span class="ow">is</span> <span class="n">down</span> <span class="n">since</span> <span class="n">epoch</span> <span class="mi">24</span><span class="p">,</span> <span class="n">last</span> <span class="n">address</span> <span class="mf">192.168.106.220</span><span class="p">:</span><span class="mi">6806</span><span class="o">/</span><span class="mi">11861</span>
</pre></div>
</div>
<p>如果想使归置组 2.5 重新在线，例如，
上面的输出告诉我们它最后由 <code class="docutils literal notranslate"><span class="pre">osd.0</span></code> 和 <code class="docutils literal notranslate"><span class="pre">osd.2</span></code> 处理，
重启这些 <code class="docutils literal notranslate"><span class="pre">ceph-osd</span></code> 将恢复那个归置组（还有其它的很多 PG ）。</p>
</section>
<section id="id9">
<h2>只有几个 OSD 接收数据<a class="headerlink" href="#id9" title="Permalink to this heading"></a></h2>
<p>如果你的集群有很多节点，但只有其中几个接收数据，检查下
存储池的归置组数量，按照 <a class="reference internal" href="../../operations/placement-groups/#rados-ops-pgs-get-pg-num"><span class="std std-ref">归置组</span></a>
文档里的方法。因为在把归置组映射到多个 OSD 的操作中，
是按照集群内 OSD 的数量来确定集群归置组数量的，
这样归置组（在此操作中，是余下的部分）数量较小时就不能分布于整个集群。
在这样的情况下，创建存储池时的归置组数量应该是 OSD 数量的若干倍，
详情见<a class="reference external" href="../../operations/placement-groups">归置组</a>。参考
<a class="reference internal" href="../../configuration/pool-pg-config-ref/#rados-config-pool-pg-crush-ref"><span class="std std-ref">存储池、归置组和 CRUSH 配置参考</span></a>
里的指导，去更改分配给各个存储池的默认归置组数量。</p>
</section>
<section id="id10">
<h2>不能写入数据<a class="headerlink" href="#id10" title="Permalink to this heading"></a></h2>
<p>如果你的集群已启动，但一些 OSD 没起来，导致不能写入数据，
确认下运行的 OSD 数量满足归置组要求的最低 OSD 数。
如果不能满足， Ceph 就不会允许你写入数据，
因为 Ceph 不能保证复制能如愿进行。
详情参见<a class="reference external" href="../../configuration/pool-pg-config-ref">存储池、归置组和 CRUSH 配置参考</a>里的
<code class="docutils literal notranslate"><span class="pre">osd</span> <span class="pre">pool</span> <span class="pre">default</span> <span class="pre">min</span> <span class="pre">size</span></code> 。</p>
</section>
<section id="id11">
<h2>归置组不一致<a class="headerlink" href="#id11" title="Permalink to this heading"></a></h2>
<p>如果 <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">health</span> <span class="pre">detail</span></code> 返回的状态是
<code class="docutils literal notranslate"><span class="pre">active</span> <span class="pre">+</span> <span class="pre">clean</span> <span class="pre">+</span> <span class="pre">inconsistent</span></code> ，
这表明可能在洗刷时遇到了错误。
用下面的命令找出不一致的归置组：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">$<span class="w"> </span>ceph<span class="w"> </span>health<span class="w"> </span>detail</span>
</pre></div></div><div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">HEALTH_ERR</span> <span class="mi">1</span> <span class="n">pgs</span> <span class="n">inconsistent</span><span class="p">;</span> <span class="mi">2</span> <span class="n">scrub</span> <span class="n">errors</span>
<span class="n">pg</span> <span class="mf">0.6</span> <span class="ow">is</span> <span class="n">active</span><span class="o">+</span><span class="n">clean</span><span class="o">+</span><span class="n">inconsistent</span><span class="p">,</span> <span class="n">acting</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">]</span>
<span class="mi">2</span> <span class="n">scrub</span> <span class="n">errors</span>
</pre></div>
</div>
<p>另外，如果你喜欢程序化的输出可以用这个命令：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">$<span class="w"> </span>rados<span class="w"> </span>list-inconsistent-pg<span class="w"> </span>rbd</span>
</pre></div></div><div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[</span><span class="s2">&quot;0.6&quot;</span><span class="p">]</span>
</pre></div>
</div>
<p>一致的状态只有一种，然而在最坏的情况下，
我们可能会遇到多个对象产生了各种各样的不一致。
假设在 PG <code class="docutils literal notranslate"><span class="pre">0.6</span></code> 里的一个名为 <code class="docutils literal notranslate"><span class="pre">foo</span></code> 的对象被截断了，
<code class="docutils literal notranslate"><span class="pre">rados</span> <span class="pre">list-inconsistent-pg</span> <span class="pre">rbd</span></code> 命令的输出可能是这样的：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">$<span class="w"> </span>rados<span class="w"> </span>list-inconsistent-obj<span class="w"> </span><span class="m">0</span>.6<span class="w"> </span>--format<span class="o">=</span>json-pretty</span>
</pre></div></div><div class="highlight-javascript notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
<span class="w">    </span><span class="s2">&quot;epoch&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">14</span><span class="p">,</span>
<span class="w">    </span><span class="s2">&quot;inconsistents&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">        </span><span class="p">{</span>
<span class="w">            </span><span class="s2">&quot;object&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">{</span>
<span class="w">                </span><span class="s2">&quot;name&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;foo&quot;</span><span class="p">,</span>
<span class="w">                </span><span class="s2">&quot;nspace&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;&quot;</span><span class="p">,</span>
<span class="w">                </span><span class="s2">&quot;locator&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;&quot;</span><span class="p">,</span>
<span class="w">                </span><span class="s2">&quot;snap&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;head&quot;</span><span class="p">,</span>
<span class="w">                </span><span class="s2">&quot;version&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">1</span>
<span class="w">            </span><span class="p">},</span>
<span class="w">            </span><span class="s2">&quot;errors&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">                </span><span class="s2">&quot;data_digest_mismatch&quot;</span><span class="p">,</span>
<span class="w">                </span><span class="s2">&quot;size_mismatch&quot;</span>
<span class="w">            </span><span class="p">],</span>
<span class="w">            </span><span class="s2">&quot;union_shard_errors&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">                </span><span class="s2">&quot;data_digest_mismatch_info&quot;</span><span class="p">,</span>
<span class="w">                </span><span class="s2">&quot;size_mismatch_info&quot;</span>
<span class="w">            </span><span class="p">],</span>
<span class="w">            </span><span class="s2">&quot;selected_object_info&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;0:602f83fe:::foo:head(16&#39;1 client.4110.0:1 dirty|data_digest|omap_digest s 968 uv 1 dd e978e67f od ffffffff alloc_hint [0 0 0])&quot;</span><span class="p">,</span>
<span class="w">            </span><span class="s2">&quot;shards&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">                </span><span class="p">{</span>
<span class="w">                    </span><span class="s2">&quot;osd&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">0</span><span class="p">,</span>
<span class="w">                    </span><span class="s2">&quot;errors&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[],</span>
<span class="w">                    </span><span class="s2">&quot;size&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">968</span><span class="p">,</span>
<span class="w">                    </span><span class="s2">&quot;omap_digest&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;0xffffffff&quot;</span><span class="p">,</span>
<span class="w">                    </span><span class="s2">&quot;data_digest&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;0xe978e67f&quot;</span>
<span class="w">                </span><span class="p">},</span>
<span class="w">                </span><span class="p">{</span>
<span class="w">                    </span><span class="s2">&quot;osd&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">1</span><span class="p">,</span>
<span class="w">                    </span><span class="s2">&quot;errors&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[],</span>
<span class="w">                    </span><span class="s2">&quot;size&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">968</span><span class="p">,</span>
<span class="w">                    </span><span class="s2">&quot;omap_digest&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;0xffffffff&quot;</span><span class="p">,</span>
<span class="w">                    </span><span class="s2">&quot;data_digest&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;0xe978e67f&quot;</span>
<span class="w">                </span><span class="p">},</span>
<span class="w">                </span><span class="p">{</span>
<span class="w">                    </span><span class="s2">&quot;osd&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">2</span><span class="p">,</span>
<span class="w">                    </span><span class="s2">&quot;errors&quot;</span><span class="o">:</span><span class="w"> </span><span class="p">[</span>
<span class="w">                        </span><span class="s2">&quot;data_digest_mismatch_info&quot;</span><span class="p">,</span>
<span class="w">                        </span><span class="s2">&quot;size_mismatch_info&quot;</span>
<span class="w">                    </span><span class="p">],</span>
<span class="w">                    </span><span class="s2">&quot;size&quot;</span><span class="o">:</span><span class="w"> </span><span class="mf">0</span><span class="p">,</span>
<span class="w">                    </span><span class="s2">&quot;omap_digest&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;0xffffffff&quot;</span><span class="p">,</span>
<span class="w">                    </span><span class="s2">&quot;data_digest&quot;</span><span class="o">:</span><span class="w"> </span><span class="s2">&quot;0xffffffff&quot;</span>
<span class="w">                </span><span class="p">}</span>

<span class="w">        </span><span class="p">}</span>
<span class="w">    </span><span class="p">]</span>
<span class="p">}</span>
</pre></div>
</div>
<p>此时，我们可以从输出里看到：</p>
<ul class="simple">
<li><p>唯一不一致的对象名为 <code class="docutils literal notranslate"><span class="pre">foo</span></code> ，并且这就是它不一致的 head 。</p></li>
<li><p>不一致分为两类：</p>
<ol class="arabic simple">
<li><p><code class="docutils literal notranslate"><span class="pre">errors</span></code>: 这些错误表明不一致性出现在分片之间，但是没说明哪个（或哪些）分片有问题。如果 <code class="docutils literal notranslate"><span class="pre">shards</span></code> 数组中有 <code class="docutils literal notranslate"><span class="pre">errors</span></code> 字段，且不为空，它会指出问题所在。</p>
<ul>
<li><p><code class="docutils literal notranslate"><span class="pre">data_digest_mismatch</span></code>: OSD.2 内读取到的副本的数字摘要与 OSD.0 和 OSD.1 的不一样。</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">size_mismatch</span></code>: OSD.2 内读取到的副本的尺寸是 0 ，而
OSD.0 和 OSD.1 说是 968 。</p></li>
</ul>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">union_shard_errors</span></code>: <code class="docutils literal notranslate"><span class="pre">shards</span></code> 数组中、所有与分片相关的错误 <code class="docutils literal notranslate"><span class="pre">errors</span></code> 的联合体。 <code class="docutils literal notranslate"><span class="pre">errors</span></code> 是个错误原因集合，
汇集了存在这类问题的分片，包括 <code class="docutils literal notranslate"><span class="pre">read_error</span></code> 等相似的错误。
以 <code class="docutils literal notranslate"><span class="pre">oi</span></code> 结尾的 <code class="docutils literal notranslate"><span class="pre">errors</span></code> 表明它是与 <code class="docutils literal notranslate"><span class="pre">selected_object_info</span></code> 的对照结果。
从 <code class="docutils literal notranslate"><span class="pre">shards</span></code> 数组里可以查到哪个分片有什么样的错误。</p>
<ul>
<li><p><code class="docutils literal notranslate"><span class="pre">data_digest_mismatch_info</span></code>: 存储在 object-info （对象信息）
里的数字签名不是 <code class="docutils literal notranslate"><span class="pre">0xffffffff</span></code>
（这个是根据 OSD.2 上的分片计算出来的）。</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">size_mismatch_info</span></code>: <code class="docutils literal notranslate"><span class="pre">object-info</span></code> 内存储的尺寸与 <code class="docutils literal notranslate"><span class="pre">OSD.2</span></code>
上读出的对象尺寸不同。后者是 <code class="docutils literal notranslate"><span class="pre">0</span></code> 。</p></li>
</ul>
</li>
</ol>
</li>
</ul>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>如果 <code class="docutils literal notranslate"><span class="pre">read_error</span></code> 出现在一个分片的 <code class="docutils literal notranslate"><span class="pre">errors</span></code> 属性里，
不一致很可能是物理存储器错误导致的。遇到这样的情况，
就要检查下那个 OSD 所用的存储驱动器。</p>
<p>在维修驱动器前，先检查下 <code class="docutils literal notranslate"><span class="pre">dmesg</span></code> 和 <code class="docutils literal notranslate"><span class="pre">smartctl</span></code> 的输出。</p>
</div>
<p>你可以用下列命令修复不一致的归置组：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>pg<span class="w"> </span>repair<span class="w"> </span><span class="o">{</span>placement-group-ID<span class="o">}</span></span>
</pre></div></div><p>例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt2:before {
  content: "# ";
}
</style><span class="prompt2">ceph<span class="w"> </span>pg<span class="w"> </span>repair<span class="w"> </span><span class="m">1</span>.4</span>
</pre></div></div><div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>此命令会用<cite>权威的</cite>副本覆盖<cite>有问题的</cite>。
根据既定规则，多数情况下 Ceph 都能从若干副本中选择正确的，
但也不是所有情况下都行得通，也会有例外。比如，
存储的数字签名可能正好丢了，
Ceph 选择权威副本时就会忽略计算出的数字签名，
总之，用此命令时小心为好。</p>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>PG ID 的格式是 <code class="docutils literal notranslate"><span class="pre">N.xxxxx</span></code> ，其中 <code class="docutils literal notranslate"><span class="pre">N</span></code> 是这个 PG
所在存储池的号码。 <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">osd</span> <span class="pre">listpools</span></code> 命令和
<code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">osd</span> <span class="pre">dump</span> <span class="pre">|</span> <span class="pre">grep</span> <span class="pre">pool</span></code> 命令可以查看存储池号码列表。</p>
</div>
<p>如果你时不时遇到时钟偏移引起的 <code class="docutils literal notranslate"><span class="pre">active</span> <span class="pre">+</span> <span class="pre">clean</span> <span class="pre">+</span> <span class="pre">inconsistent</span></code> 状态，
最好在监视器主机上配置 peer 角色的
<a class="reference external" href="https://en.wikipedia.org/wiki/Network_Time_Protocol">NTP</a> 服务。
配置细节可参考<a class="reference external" href="http://www.ntp.org">网络时间协议</a>和 Ceph
<a class="reference internal" href="../../configuration/mon-config-ref/#mon-config-ref-clock"><span class="std std-ref">时钟选项</span></a>。</p>
<section id="pg">
<h3>有关修复 PG 的更多信息<a class="headerlink" href="#pg" title="Permalink to this heading"></a></h3>
<p>Ceph 会存储和更新集群内存储着的对象的校验和。
在洗刷一个 PG 时， lead OSD 会尝试从其副本中选择一个权威副本。
各种可能的情况里只有一种是一致的。
执行深度洗刷后， Ceph 会计算从磁盘读出的每个对象的校验和，
并将其与之前记录的校验和进行比对。
如果当前校验和与之前记录的校验和不匹配，
那么这个不匹配将被视为不一致。对于多副本存储池来说，
对象的任何副本与权威副本的校验和不匹配都意味着存在不一致。
发现这些不一致会导致 PG 的状态被设置为不一致（ <code class="docutils literal notranslate"><span class="pre">inconsistent</span></code> ）。</p>
<p><code class="docutils literal notranslate"><span class="pre">pg</span> <span class="pre">repair</span></code> 命令试图修复各种类型的不一致。
当 <code class="docutils literal notranslate"><span class="pre">pg</span> <span class="pre">repair</span></code> 发现不一致的 PG 时，
它会尝试用权威副本的签名覆盖不一致副本的签名。
当 <code class="docutils literal notranslate"><span class="pre">pg</span> <span class="pre">repair</span></code> 在多副本存储池中发现不一致副本时，
它会将不一致副本标记为丢失。对于多副本存储池，
恢复不在 <code class="docutils literal notranslate"><span class="pre">pg</span> <span class="pre">repair</span></code> 的范围内。</p>
<p>对于纠删码存储池和 BlueStore 存储池，如果
<code class="docutils literal notranslate"><span class="pre">osd_scrub_auto_repair</span></code> （默认为 <code class="docutils literal notranslate"><span class="pre">false</span></code> ）设置为 <code class="docutils literal notranslate"><span class="pre">true</span></code> ，
而且发现的错误个数不超过 <code class="docutils literal notranslate"><span class="pre">osd_scrub_auto_repair_num_errors</span></code>
（默认为 <code class="docutils literal notranslate"><span class="pre">5</span></code> ）， Ceph 就会自动执行修复。</p>
<p><code class="docutils literal notranslate"><span class="pre">pg</span> <span class="pre">repair</span></code> 命令不能解决所有问题。
当发现 PG 存在不一致时，Ceph 也不会自动修复 PG。</p>
<p>RADOS 对象或 omap 的校验和并非总是可用。
校验和是以增量方式计算的。如果一个对象副本不是顺序更新的，
更新时的写入操作会改变对象并使其校验和失效。
在重新计算校验和时，不会读取整个对象。
即使无法获得校验和， <code class="docutils literal notranslate"><span class="pre">pg</span> <span class="pre">repair</span></code> 命令也能进行修复，
比如在使用 Filestore 时。使用多副本 Filestore 存储池
的用户可能更喜欢手动修复而不是 <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">pg</span> <span class="pre">repair</span></code> 。</p>
<p>这部分材料与 Filestore 有关，而与 BlueStore 无关，因为 BlueStore
有它自己的内部校验和。记录的校验和与计算出的校验和相匹配
并不能证明哪一份副本确实是权威的。
如果没有可用的校验和， <code class="docutils literal notranslate"><span class="pre">pg</span> <span class="pre">repair</span></code> 会倾向于主 PG 内的数据，
但它可能不是未损坏的副本。由于存在这种不确定性，
所以发现不一致时有必要人工介入。
人工介入有时还得使用 <code class="docutils literal notranslate"><span class="pre">ceph-objectstore-tool</span></code> 工具。</p>
</section>
<section id="id13">
<h3>PG 修复实战<a class="headerlink" href="#id13" title="Permalink to this heading"></a></h3>
<p><a class="reference external" href="https://ceph.io/geen-categorie/ceph-manually-repair-object/">https://ceph.io/geen-categorie/ceph-manually-repair-object/</a> - 这个网页
上有一个 PG 修复的实战记录。如果你从来没修复过，
现在想试试，建议先读一下。</p>
</section>
</section>
<section id="active-clean">
<h2>纠删编码的归置组不是 active+clean<a class="headerlink" href="#active-clean" title="Permalink to this heading"></a></h2>
<p>CRUSH 找不到足够多的 OSD 映射到某个 PG 时，它会显示为
<code class="docutils literal notranslate"><span class="pre">2147483647</span></code> ，意思是 <code class="docutils literal notranslate"><span class="pre">ITEM_NONE</span></code> 或 <code class="docutils literal notranslate"><span class="pre">no</span> <span class="pre">OSD</span> <span class="pre">found</span></code> ，例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">8</span><span class="p">,</span><span class="mi">2147483647</span><span class="p">,</span><span class="mi">7</span><span class="p">,</span><span class="mi">4</span><span class="p">]</span>
</pre></div>
</div>
<section id="id14">
<h3>OSD 不够多<a class="headerlink" href="#id14" title="Permalink to this heading"></a></h3>
<p>如果 Ceph 集群仅有 8 个 OSD ，但是纠删码存储池需要 9 个，
集群就会显示 “Not enough OSDs” 。这时候，
你还可以另外创建一个需要较少 OSD 的纠删码存储池，
命令格式如下constraints：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>erasure-code-profile<span class="w"> </span><span class="nb">set</span><span class="w"> </span>myprofile<span class="w"> </span><span class="nv">k</span><span class="o">=</span><span class="m">5</span><span class="w"> </span><span class="nv">m</span><span class="o">=</span><span class="m">3</span></span>
<span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>erasurepool<span class="w"> </span>erasure<span class="w"> </span>myprofile</span>
</pre></div></div><p>或者新增一个 OSD ，这个 PG 会自动用上的。</p>
</section>
<section id="id15">
<h3>CRUSH 条件不能满足<a class="headerlink" href="#id15" title="Permalink to this heading"></a></h3>
<p>即使集群拥有足够多的 OSD ， CRUSH 规则的强制条件仍有可能无法满足。
假如有 10 个 OSD 分布于两个主机上，
而 CRUSH 规则要求相同归置组不得使用位于同一主机的两个 OSD ，
这样映射就会失败，因为只能找到两个 OSD 。
你可以从规则里查看（ dump ）必要条件：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>crush<span class="w"> </span>rule<span class="w"> </span>ls</span>
</pre></div></div><div class="highlight-default notranslate"><div class="highlight"><pre><span></span>[
    &quot;replicated_rule&quot;,
    &quot;erasurepool&quot;]
$ ceph osd crush rule dump erasurepool
{ &quot;rule_id&quot;: 1,
  &quot;rule_name&quot;: &quot;erasurepool&quot;,
  &quot;type&quot;: 3,
  &quot;steps&quot;: [
        { &quot;op&quot;: &quot;take&quot;,
          &quot;item&quot;: -1,
          &quot;item_name&quot;: &quot;default&quot;},
        { &quot;op&quot;: &quot;chooseleaf_indep&quot;,
          &quot;num&quot;: 0,
          &quot;type&quot;: &quot;host&quot;},
        { &quot;op&quot;: &quot;emit&quot;}]}
</pre></div>
</div>
<p>可以这样解决此问题，创建新存储池，其内的 PG 允许多个 OSD
位于同一主机，命令如下：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>erasure-code-profile<span class="w"> </span><span class="nb">set</span><span class="w"> </span>myprofile<span class="w"> </span>crush-failure-domain<span class="o">=</span>osd</span>
<span class="prompt1">ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>erasurepool<span class="w"> </span>erasure<span class="w"> </span>myprofile</span>
</pre></div></div></section>
<section id="id16">
<h3>CRUSH 过早中止<a class="headerlink" href="#id16" title="Permalink to this heading"></a></h3>
<p>假设集群拥有的 OSD 足以映射到 PG
（比如有 9 个 OSD 和一个纠删码存储池的集群，
每个 PG 需要 9 个 OSD ）， CRUSH 仍然有可能在找到映射前就中止了。可以这样解决：</p>
<ul class="simple">
<li><p>降低纠删存储池内 PG 的要求，让它使用较少的 OSD
（需创建另一个存储池，
因为纠删码配置不支持动态修改）。</p></li>
<li><p>向集群添加更多 OSD （无需修改纠删存储池，
它会自动回到清洁状态）。</p></li>
<li><p>通过手工打造的 CRUSH 规则，让它多试几次以找到合适的映射。
把 <code class="docutils literal notranslate"><span class="pre">set_choose_tries</span></code> 设置得高于默认值即可。</p></li>
</ul>
<p>从集群中提取出 crushmap 之后，应该先用 <code class="docutils literal notranslate"><span class="pre">crushtool</span></code> 校验一下是否有问题，这样你的试验就无需触及 Ceph 集群，只要在一个本地文件上测试即可：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">$<span class="w"> </span>ceph<span class="w"> </span>osd<span class="w"> </span>crush<span class="w"> </span>rule<span class="w"> </span>dump<span class="w"> </span>erasurepool</span>
</pre></div></div><div class="highlight-default notranslate"><div class="highlight"><pre><span></span>{ &quot;rule_id&quot;: 1,
  &quot;rule_name&quot;: &quot;erasurepool&quot;,
  &quot;type&quot;: 3,
  &quot;steps&quot;: [
        { &quot;op&quot;: &quot;take&quot;,
          &quot;item&quot;: -1,
          &quot;item_name&quot;: &quot;default&quot;},
        { &quot;op&quot;: &quot;chooseleaf_indep&quot;,
          &quot;num&quot;: 0,
          &quot;type&quot;: &quot;host&quot;},
        { &quot;op&quot;: &quot;emit&quot;}]}
$ ceph osd getcrushmap &gt; crush.map
got crush map from osdmap epoch 13
$ crushtool -i crush.map --test --show-bad-mappings \
   --rule 1 \
   --num-rep 9 \
   --min-x 1 --max-x $((1024 * 1024))
bad mapping rule 8 x 43 num_rep 9 result [3,2,7,1,2147483647,8,5,6,0]
bad mapping rule 8 x 79 num_rep 9 result [6,0,2,1,4,7,2147483647,5,8]
bad mapping rule 8 x 173 num_rep 9 result [0,4,6,8,2,1,3,7,2147483647]
</pre></div>
</div>
<p>其中 <code class="docutils literal notranslate"><span class="pre">--num-rep</span></code> 是纠删码 CRUSH 规则所需的 OSD 数量，
<code class="docutils literal notranslate"><span class="pre">--rule</span></code> 是 <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">osd</span> <span class="pre">crush</span> <span class="pre">rule</span> <span class="pre">dump</span></code> 命令结果中
<code class="docutils literal notranslate"><span class="pre">rule_id</span></code> 字段的值。此测试会尝试映射一百万个值
（即 <code class="docutils literal notranslate"><span class="pre">[--min-x,--max-x]</span></code> 所指定的范围），
且必须至少显示一个坏映射；如果它没有任何输出，
说明所有映射都成功了，你可以就此打住：
问题的根源不在这里。</p>
<section id="set-choose-tries">
<h4>修改 set_choose_tries 的值<a class="headerlink" href="#set-choose-tries" title="Permalink to this heading"></a></h4>
<ol class="arabic">
<li><p>反编译 crush 图后，你可以手动编辑其 CRUSH 规则：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">crushtool<span class="w"> </span>--decompile<span class="w"> </span>crush.map<span class="w"> </span>&gt;<span class="w"> </span>crush.txt</span>
</pre></div></div></li>
<li><p>并把下面这行加进规则：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">step</span> <span class="n">set_choose_tries</span> <span class="mi">100</span>
</pre></div>
</div>
<p>然后 <code class="docutils literal notranslate"><span class="pre">crush.txt</span></code> 文件内的这部分大致如此：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">rule</span> <span class="n">erasurepool</span> <span class="p">{</span>
        <span class="nb">id</span> <span class="mi">1</span>
        <span class="nb">type</span> <span class="n">erasure</span>
        <span class="n">step</span> <span class="n">set_chooseleaf_tries</span> <span class="mi">5</span>
        <span class="n">step</span> <span class="n">set_choose_tries</span> <span class="mi">100</span>
        <span class="n">step</span> <span class="n">take</span> <span class="n">default</span>
        <span class="n">step</span> <span class="n">chooseleaf</span> <span class="n">indep</span> <span class="mi">0</span> <span class="nb">type</span> <span class="n">host</span>
        <span class="n">step</span> <span class="n">emit</span>
<span class="p">}</span>
</pre></div>
</div>
</li>
<li><p>然后编译、并再次测试：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">crushtool<span class="w"> </span>--compile<span class="w"> </span>crush.txt<span class="w"> </span>-o<span class="w"> </span>better-crush.map</span>
</pre></div></div></li>
<li><p>所有映射都成功时，
用 <code class="docutils literal notranslate"><span class="pre">crushtool</span></code> 的 <code class="docutils literal notranslate"><span class="pre">--show-choose-tries</span></code> 选项能看到所有成功映射的尝试次数直方图，
如下面的例子：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1"><span class="w">  </span>crushtool<span class="w"> </span>-i<span class="w"> </span>better-crush.map<span class="w"> </span>--test<span class="w"> </span>--show-bad-mappings<span class="w"> </span><span class="se">\</span>
<span class="w">   </span>--show-choose-tries<span class="w"> </span><span class="se">\</span>
<span class="w">   </span>--rule<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="se">\</span>
<span class="w">   </span>--num-rep<span class="w"> </span><span class="m">9</span><span class="w"> </span><span class="se">\</span>
<span class="w">   </span>--min-x<span class="w"> </span><span class="m">1</span><span class="w"> </span>--max-x<span class="w"> </span><span class="k">$((</span><span class="m">1024</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="m">1024</span><span class="k">))</span></span>
<span class="prompt1">...</span>
<span class="prompt1"><span class="m">11</span>:<span class="w">        </span><span class="m">42</span></span>
<span class="prompt1"><span class="m">12</span>:<span class="w">        </span><span class="m">44</span></span>
<span class="prompt1"><span class="m">13</span>:<span class="w">        </span><span class="m">54</span></span>
<span class="prompt1"><span class="m">14</span>:<span class="w">        </span><span class="m">45</span></span>
<span class="prompt1"><span class="m">15</span>:<span class="w">        </span><span class="m">35</span></span>
<span class="prompt1"><span class="m">16</span>:<span class="w">        </span><span class="m">34</span></span>
<span class="prompt1"><span class="m">17</span>:<span class="w">        </span><span class="m">30</span></span>
<span class="prompt1"><span class="m">18</span>:<span class="w">        </span><span class="m">25</span></span>
<span class="prompt1"><span class="m">19</span>:<span class="w">        </span><span class="m">19</span></span>
<span class="prompt1"><span class="m">20</span>:<span class="w">        </span><span class="m">22</span></span>
<span class="prompt1"><span class="m">21</span>:<span class="w">        </span><span class="m">20</span></span>
<span class="prompt1"><span class="m">22</span>:<span class="w">        </span><span class="m">17</span></span>
<span class="prompt1"><span class="m">23</span>:<span class="w">        </span><span class="m">13</span></span>
<span class="prompt1"><span class="m">24</span>:<span class="w">        </span><span class="m">16</span></span>
<span class="prompt1"><span class="m">25</span>:<span class="w">        </span><span class="m">13</span></span>
<span class="prompt1"><span class="m">26</span>:<span class="w">        </span><span class="m">11</span></span>
<span class="prompt1"><span class="m">27</span>:<span class="w">        </span><span class="m">11</span></span>
<span class="prompt1"><span class="m">28</span>:<span class="w">        </span><span class="m">13</span></span>
<span class="prompt1"><span class="m">29</span>:<span class="w">        </span><span class="m">11</span></span>
<span class="prompt1"><span class="m">30</span>:<span class="w">        </span><span class="m">10</span></span>
<span class="prompt1"><span class="m">31</span>:<span class="w">         </span><span class="m">6</span></span>
<span class="prompt1"><span class="m">32</span>:<span class="w">         </span><span class="m">5</span></span>
<span class="prompt1"><span class="m">33</span>:<span class="w">        </span><span class="m">10</span></span>
<span class="prompt1"><span class="m">34</span>:<span class="w">         </span><span class="m">3</span></span>
<span class="prompt1"><span class="m">35</span>:<span class="w">         </span><span class="m">7</span></span>
<span class="prompt1"><span class="m">36</span>:<span class="w">         </span><span class="m">5</span></span>
<span class="prompt1"><span class="m">37</span>:<span class="w">         </span><span class="m">2</span></span>
<span class="prompt1"><span class="m">38</span>:<span class="w">         </span><span class="m">5</span></span>
<span class="prompt1"><span class="m">39</span>:<span class="w">         </span><span class="m">5</span></span>
<span class="prompt1"><span class="m">40</span>:<span class="w">         </span><span class="m">2</span></span>
<span class="prompt1"><span class="m">41</span>:<span class="w">         </span><span class="m">5</span></span>
<span class="prompt1"><span class="m">42</span>:<span class="w">         </span><span class="m">4</span></span>
<span class="prompt1"><span class="m">43</span>:<span class="w">         </span><span class="m">1</span></span>
<span class="prompt1"><span class="m">44</span>:<span class="w">         </span><span class="m">2</span></span>
<span class="prompt1"><span class="m">45</span>:<span class="w">         </span><span class="m">2</span></span>
<span class="prompt1"><span class="m">46</span>:<span class="w">         </span><span class="m">3</span></span>
<span class="prompt1"><span class="m">47</span>:<span class="w">         </span><span class="m">1</span></span>
<span class="prompt1"><span class="m">48</span>:<span class="w">         </span><span class="m">0</span></span>
<span class="prompt1">...</span>
<span class="prompt1"><span class="m">102</span>:<span class="w">         </span><span class="m">0</span></span>
<span class="prompt1"><span class="m">103</span>:<span class="w">         </span><span class="m">1</span></span>
<span class="prompt1"><span class="m">104</span>:<span class="w">         </span><span class="m">0</span></span>
<span class="prompt1">...</span>
</pre></div></div><p>有 42 个归置组需 11 次重试、 44 个归置组需 12 次重试，以此类推。这样，重试的最高次数就是防止坏映射的最低值，也就是
<code class="docutils literal notranslate"><span class="pre">set_choose_tries</span></code> 的取值（即上面输出中的 103 ，因为任意归置组成功映射的重试次数都没有超过 103 ）。</p>
</li>
</ol>
</section>
</section>
</section>
</section>



<div id="support-the-ceph-foundation" class="admonition note">
  <p class="first admonition-title">Brought to you by the Ceph Foundation</p>
  <p class="last">The Ceph Documentation is a community resource funded and hosted by the non-profit <a href="https://ceph.io/en/foundation/">Ceph Foundation</a>. If you would like to support this and our other efforts, please consider <a href="https://ceph.io/en/foundation/join/">joining now</a>.</p>
</div>


           </div>
           
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../troubleshooting-osd/" class="btn btn-neutral float-left" title="OSD 故障排除" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../log-and-debug/" class="btn btn-neutral float-right" title="日志记录和调试" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).</p>
  </div>

   

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>